N94-11427 

SOFTWARE PROCESS ASSESSMENTS 


Sharon E. Miller 
George T. Tucker 
Anthony J. Verducci, Jr. 

AT&T Bell Laboratories, USA 

Software process assessments (SPA’s) are part of an on- 
going program of continuous quality improvement in 
AT&T. Their use has been found to be very beneficial by 
software development organizations in identifying the 
issues facing the organization and the actions required to 
increase both quality and productivity in the organization. 


INTRODUCTION 


The AT&T Software Process Assessment program is part of the overall software qua lty 
improvement activities in many of the software development organizations across the 
company. Assessments are a service provided to a project or organization for the 
purpose of baselining the processes used for software development and developing an 
action plan for improvement activities. In addition, the data collected over time is use 
to identify corporate strengths and weaknesses and to drive corporate level efforts aimed 
at improving the software development environment. 


AT&T has chosen to implement its assessment program using a trained assessment team 
that is independent of the software development organization. The benefits of this 
approach will be discussed in conjunction with describing the experiences of customers 
within AT&T who have had one or more assessments. 


CUSTOMER EXPECTATIONS 


Customers have a number of expectations about an assessment, the process, and the 
findings, and meeting these expectations is critical to the success of any individual 
assessment and to the assessment program. Many of these expectations are ones t at 
customers can articulate at the outset. Others, however, are implicit expectations that 
experience has shown are important to customers, but that are not recognized by 
customers as being important until the process is completed. The most important 
expectations are listed below: 
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o The assessment team must fulfill its commitments in a timely fashion, 
o The assessment must be independent and objective, 

o The assessment must accurately identify the organization's strengths and weaknesses, 
o The feedback must be professional and non-threatening, 
o The results must be confidential, and 

o The findings and recommendations must include a roadmap for follow-on action plans. 


Fulfilling Commitments 


An assessment is a significant undertaking for an organization in terms of cost and effort. 
The average assessment for a moderately sized project is approximately 30 staff days of 
time for the assessment team and 30 staff days of time for the project. As such, the 
assessment must be planned and managed, as would any project task. Each assessment is 
assigned a project leader from the assessment team, who is responsible for all of the 
customer interfaces, including establishing the schedules, working the details of the 
assessment implementation, and ensuring all assessment team commitments are fulfilled. 
Likewise, a project coordinator from the organization being assessed is identified to work 
with the assessment leader to coordinate all the logistics, such as selecting participants, 
getting meeting rooms, sending out meeting notices, and ensuring all project 
commitments are fulfilled. 


To facilitate this process, the assessment leader meets with the project coordinator, 
provides the coordinator with a checklist, and makes sure the coordinator clearly 
understands what the assessment team will be doing and what the project must 
do throughout the life-cycle of the assessment. The assessment leader then assumes the 
responsibility for following up at each step of the process to ensure all activities are 
performed as agreed. 


Independence and Objectivity 


While it is possible for an organization to perform a self-assessment, experience in 
AT&T has shown that having a trained assessment team not associated with the project is 
more effective. The objectivity is perceived to be higher when the assessment team has 
no vested interest in the results or the subsequent changes that are recommended for 
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improvement. Even though the data represents the views of the members of the project 
team participating in the assessment, management tends to be more accepting of the 
results and the recommendations when they come from an independent organization. 
Also, an independent team can deliver a "bad news" message more easily, particularly 1 
it involves issues like management style, communication or decision making. 


Identification of Strengths and Weaknesses 


Key to the success of the assessment from the customer perspective is the ability of the 
assessment team to identify, from the data collected, the key strengths and weaknesses in 
the organization. The customer will not be satisfied with the assessment results if this is 
not done accurately. Thus, it is important for the assessment team to adequately explore 
all the pertinent issues during the data collection period, so that each strength or 
weakness can be supported by fact. 


Having an assessment performed as a mechanism solely to understand where to focus 
quality improvement efforts carries an implicitly negative connotation. The expectation 
is that the organization will be given a laundry list of deficiencies and recommendations 
to address them. Experience has again shown that the approach taken in AT&T, in 
which the assessment balances the feedback to include both strengths and weaknesses, 
meets customer expectations for learning their deficiencies, but does so in a much more 

positive fashion. 


Another reason for balancing the strengths and weaknesses is to reinforce the things a 
project is doing very well, so it does not lose sight of them or stop doing them while 
focusing attention on efforts to improve the deficiencies. 


Providing Feedback 


As mentioned above, the feedback to an organization must be handled professionally, so 
that the overall result is a positive one. While the goal is to have an organization take 
action on deficiencies uncovered during the process, it is equally important to ensure that 
the organization buys into the feedback. This means having concrete data to support the 
findings and using lots of specific project examples when discussing the findings. The 
assessment team must be adequately prepared to discuss all aspects of the findings m 
detail to ensure credibility and acceptance of the results. 
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Maintaining Confidentiality 


Another very critical success factor is the confidentiality of the data. This confidentiality 
occurs at two levels. First, the team from the project that provides the data to the 
assessment team must feel confident that their individual comments will not be 
disclosed to management. The assessment team needs to address this issue before the 
data is collected and honor this commitment to confidentiality throughout the process. 


The second level of confidentiality is at the project level. The data collected and the 
resulting feedback belong to the project and are considered to be private. The assessment 
team must again, at the start of the process, ensure that the project coordinator 
understands that other than using the data to determine corporate averages and trends, the 
data will not be released to anyone but the project coordinator without the project or 
organization’s permission. This agreement must be scrupulously honored by the 
assessment team. 


Developing Action Plans 


The real work begins when the assessment is completed and the project must undertake 
the planning and implementation of quality improvement activities. While these 
activities are generally beyond the scope of the assessment, projects have varying 
expectations as to how this will be managed. Several alternative approaches have been 
used within AT&T. 


A first, and common approach, is for the project to set up quality or process 
improvement teams to address the assessment recommendations. These teams will have 
the responsibility to examine the areas for improvement, to formulate a plan of action, 
and to oversee the implementation of the process changes. The implementation of the 
process changes may be undertaken on a trial basis or applied to the entire organization, 
but each change must be monitored for a period of time to ensure that the expected 
improvements are in fact occurring. 


A second approach taken by some projects is to ask the assessment team to participate 
with project members in the formulation of the action plan. It is important to note that 
the participation must be consultative and the project must clearly accept the 
responsibility for the ownership of this activity and the resulting process changes. It is 
not within the power of the assessment team to drive changes within an organization. 
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A third approach being tried is to assign a process engineer, who is an experienced 
software consultant, to the project at the completion of the assessment. The assessment 
results are provided to a process engineer who then works with the project on an ongoing 
basis as they develop and implement process changes. An advantage to this approach is 
the process engineer is dedicated to the project, while at the same time has access to a 
wide range of other experts, practices, and tools that can be used to help the project. 


CUSTOMER EXPERIENCES 


AT&T has done nearly 100 assessments since 1987 using one or two widely accepted 
instruments, the Software Engineering Institute’s (SEI) Capability Maturity Model/1/ and 
the Software Productivity Research* (SPR) CHECKPOINT™ questionnaire/2/. The SPR 
instrument was modified for use in AT&T and has been used in all the assessments. The 
SEI questionnaire is unmodified and has been used in roughly 40 assessments since mid 
1990. Because the two instruments have a different focus, the SPR looking primarily at 
detailed process effectiveness and the SEI looking at overall process management, they 
are administered differently. 


SEI Consensus Meeting 


As indicated above, the SEI questionnaire is management oriented, so the organization is 
asked to convene a meeting(s) of their management team(s) and to have the team(s) reach 
consensus on each question in the questionnaire. Most management teams find this 
meeting is extremely valuable because of the discussions that result from trying to reach 
consensus on how different activities are being managed in the organization. 


Another valuable use of this data is that it provides die management perspective on the 
software development process, and this perspective is not always the same as that of the 
software developers. For example, in one assessment, the management team 
indicated that the organization had documented methods and standards for the software 
development process, but when the developers were interviewed, they indicated that, 
while there was a document, the use of the methods and standards was optional. 


1 CHECKPOINT is a trademark of Software Productivity Reasearch, Inc., a company 
founded and headed by Capers Jones, a recognized expert in the area of software quality 

and productivity. 
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SPR Consensus Meeting 

The AT&T version of the SPR questionnaire contains approximately 300 questions. 

The answers to the questions, which are generally on a 1-5 scale, indicate how the 
organization is doing relative to the software industry for that particular area. For 
example, if the sample question shown below were answered, a score in the 1 to 2.5 
range would indicate the project is leading edge with respect to the software 
defect data collection function, a score in the 2.6 to 3.4 range, would indicate the project 
is at industry norm, and a score 3.5 or greater would indicate the project is at high risk 
relative to the software industry as a whole. The question structures are the result of 
Capers Jones’ extensive experience in the software field. 


Software defect data collection?: 

1) Defect data collected from requirements onward 

2) Defect data collected from design onward 

3) Defect data collected from testing onward 

4) Intermittent or partial defect data collection 

5) No defect data collection 


In the AT&T process, representatives from the software development staff, not generally 
managers, are asked to fill out the questionnaire, and to then attend a one day meeting, 
during which the group is expected to reach a consensus answer on each question. All 
major software development life cycle activities are represented, including requirements, 
architecture, design, implementation, test, product support, and user documentation. To 
help the group reach consensus, the assessment team facilitates a discussion of each 
question. This meeting generates a great deal of discussion among the participants and 
provides a great deal of insight into the organization. 


Most groups find that the meeting is informative, and often it is the first time people 
representing different functional areas have the chance to understand the issues faced by 
others. For example, it may be the first time the person responsible for 
preparing customer documentation and the software developers have a meaningful 
discussion on the documentation process and its effectiveness. 
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Hard Data 


In addition to the data collected through the two questionnaires, there is additional data 
requested of a project. This data is referred to as hard data because it is quantitative in 
nature, asking for things such as staff effort by development phase, start and stop times 
for development phases, defect counts, etc. In general, a single person in the 
organization is asked to collect this data. This data is used to calculate productivity for 
the project and to validate information from the SPR session. 


Feedback Session 


After the data has been collected and analyzed, the project is invited to a feedback 
session to hear the results of the assessment. Experience has shown that the most 
successful assessments are those in which top management has been involved 
and supportive. It is especially important that top management is present at the feedback 
session and shows a commitment to the improvement process. One of several vehicles 
used to give a project a snapshot of the effectiveness of their software processes is shown 
in Figure 1 . This is called a kiviat or spider chart. 


The kiviat graphically depicts how a project is doing in each of eight categories, using 
the same 1-5 scale discussed previously. 10-20 questions from the SPR questionnaire are 
associated with each of the eight categories. The categories are defined below. The 
average scores for these questions are then plotted to form the kiviat. In this 
representation, the closer a score is to the center of the chart, the better the project is 
doing in that particular category. In Figure 1, for example, this hypothetical project is 
doing particularly well in the customer focus, project team variables, tools, physical 
environment, and metrics areas, and is at industry norm in the project management and 
methodologies areas. 


Customer focus includes questions that deal with the level of customer involvement in 
the various stages of product development, including requirements, prototyping, 
customer documentation, and acceptance testing. 

Project management looks at the experience of the management team on similar projects, 
the effectiveness of estimation and scheduling, and other project management issues. 

Project team variables assesses the experience and training of the staff on the project. 

Tools looks at the effectiveness of the coding and non-coding tools used by the project. 
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Quality focus assesses the quality assurance activities used throughout the life cycle of 
the development effort. 

Methodologies looks at the effectiveness of the methods used in the development 
activities, from requirements and design methods to testing methods. 

Physical environment asks about the development and target hardware, the physical 
office space, and staff support issues. 

Metrics looks at the type of data that is collected and used by the project. 



Hypothetical Project Kiviat Chart 
Figure 1 
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In the feedback session, the project strengths and weaknesses are discussed m the context 
of the 8 kiviat categories. Areas that scored in the average range are not generally 
discussed, because the intent is to reinforce the strengths and to encourage improving the 
weaknesses. Feedback from projects assessed indicates that the process usually 
identifies the most significant strengths and weaknesses in an accurate and objective way. 
Feedback also indicates that, in general, projects are not very surprised at the findings. 

Another positive finding is that all the projects assessed indicate a desire to change as a 
result of the findings, and begin to address the recommendations immediately. Also 
most projects indicate a desire to undergo a reassessment in 18-24 months to gauge how 
well their improvement program has worked. 


Action Plans 


The most important activity following an assessment is the preparation of an action plan 
to address the assessment findings. The three options for preparing action plans were 
discussed earlier. Within AT&T the most common approach has been for the projects to 
tackle the action planning and improvement activities using project personnel. A 
positive aspect of taking this approach is that the project takes full ownership of the 
planning and the improvement activities, and to the extent that management follows 
through with support during the implementation phase, it has a high success rate in terms 
of lasting improvement. A disadvantage of this approach is it may not take fu 
advantage of the resources available across a company as large as AT&T, in terms o 
using solutions that have been tried and proven in other organizations. 


To help identify the broadest set of solutions possible, the assessment team, if the 
organization desires, will prepare a proposal for improvement at the conclusion of the 
assessment. This proposal identifies resources, tools, and techniques that the 
organization might use in implementing an improvement program. Members of the 
assessment team are also available for on-going consultation with the organization. 


AT&T also has a program that publishes and supports "Best Current Practices ' which 
helps to disseminate good development practices among organizations. This program is 
supported with documentation, training, and a "best practice owner," an expert on that 
particular practice, who is available for consulting with projects. Additionally, this 
program fosters the sharing of personal experiences among projects, which this has been 
found to be one of its greatest benefits. 


2 Activities that have been identified as beneficial are documented and supported through 
documentation and training. Examples of best current practices are code inspections and root cause 

analysis. 
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Reassessing an Organization 


implementing an improvement program takes time and patience, and the time required to 
see lasting benefits of the changes varies with the improvement being undertaken. For 
example, a project that implements an inspection program for requirements and design 
documents is likely to see tangible benefits of this program early in the next development 
cycle, while a project that needs to implement a formal estimation process will not be 
able to judge the benefit of the process until data has been collected and compared to the 
estimates over a full software development cycle. 

In AT&T's experience, 18-24 months is a reasonable time between assessments. In this 
time period most projects will complete a full development cycle, and will have enough 
data on the effectiveness of the changes implemented to make a reassessment 
meaningful. It is important in a software assessment that the processes being discussed 
are those in use, not those that are planned for a future development cycle. 

As an example, one AT&T project has participated in the assessment program since 
1987, having had assessments in 1987, 1989, and 1991. Following each assessment, the 
project put in place plans to address the weaknesses identified, and in each subsequent 
assessment, very clear improvement was noted. Projects with results similar to this are 
understandably proud of their progress, and generally willing to share their success 
stories with other projects just starting an improvement program. Figure 2 below is an 
example of the type of progress that can be made over a four year period in terms of a pie 
chart. The pie chart is an overall measure of the number of areas where the organization 
is leading edge, at industry norm or at high risk, relative to the questions in the SPR 
questionnaire. 



Figure 2 
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SUMMARY 


This paper has described the use of the software process assessment program m AT&T, 
and demonstrated its value as an ongoing quality improvement tool for the software 
development community. AT&T will continue to encourage software organizations to 
participate in this program and to use the results to ensure that all software products 

achieve the highest levels of quality. Additionally, since good processes have been 

shown to correlate with higher levels of productivity and shorter intervals, it is in the best 
interest of the organization to strive for leading edge development practices. 
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SOFTWARE PROCESS ASSESSMENT 
OBJECTIVES 

• Identify an organization’s strengths and weaknesses 

• Baseline the organization’s processes against industry standards 

• Compare the organization to others in AT&T 

• Prepare a roadmap for improvement activities 
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SOFTWARE PROCESS ASSESSMENT 


METHODS 


Basis: 


Focus: 


SPR 

Capers Jones CHECKPOINT™ 
Questionnaire 

Software Development 
Process Effectiveness 


SEI 

Watts Humphrey Capability 
Maturity Model 

Software Development 
Process Management 


AT&T Projects - 1Q0 since 1987 - 40 since 1989 

Assessed: 


SOFTWARE PROCESS ASSESSMENT 


PROCESS 


• Pl annin g 

— Agreement on scope, schedule, cost 

• Execution 

— Questionnaires 

• SEI by management 

• SPR by development staff 

— - Consensus meeting for SEI questionnaire 

— Facilitated meeting for SPR questionnaire 


SEL-92-004 page 124 



SOFTWARE PROCESS ASSESSMENT 


PROCESS (Continued) 


Execution (Continued) 

— Validation of SEI answers 

— Synthesis of results 

— Feedback to project 
Follow-on 

— Proposal (optional) 


SOFTWARE PROCESS ASSESSMENT 

CUSTOMER EXPECTATIONS 

Commitments fulfilled 

Independence and objectivity maintained 

Strengths and weaknesses identified 

Feedback provided 

Confidentiality ensured 

Action planning available 



SOFTWARE PROCESS ASSESSMENT 


CUSTOMER FEEDBACK 

• Kiviat profile 

• Strengths 

• Weaknesses 

• Project comparisons 

• Open-ended questions feedback 


SOFTWARE PROCESS ASSESSMENT 

CUSTOMER FEEDBACK (Continued) 

• SEI maturity evaluation 

• Project goals & constraints 

• Overall assessment findings 

• Recommendations 
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SOFTWARE PROCESS ASSESSMENT 
SOFTWARE DEVELOPMENT PROFILE 

CUMatai 


* 



SOFTWARE PROCESS ASSESSMENT 
PIE CHART COMPARISONS 


WaBAhaad WtRAhaad 

41 % 33 % 
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I! 


ROLE IN AN IMPROVEMENT PROGRAM 


SOFTWARE PROCESS ASSESSMENT 
ACTION PLAN 

l 

IMPLEMENTATION PLANS 



Measurement of 
Benefits (Reassessment) 
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